Papers with T5 model
A Thorough Evaluation of Task-Specific Pretraining for Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Previous work has used task-agnostic pretraining methods like masked language models or corrupted span prediction to improve performance on downstream tasks. |
| Approach: | They propose to use a task-agnostic pretraining to improve on low-resource tasks. |
| Outcome: | The proposed model can predict extracted gap sentences on summarization with a low resource and zero shot setup. |
MedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries (2023.acl-srw)
Copied to clipboard
| Challenge: | Clinical texts contain important temporal information, such as medication start and end dates, appointment dates, and diagnosis dates. |
| Approach: | They propose to use prompt-based learning and fine-tuning to classify temporal relations between treatments and hospitalisation periods in discharge summaries. |
| Outcome: | The proposed method identifies whether a treatment was administered between the time of admission and discharge from the hospital. |
AraT5: Text-to-Text Transformers for Arabic Language Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models that convert text-based language problems into text-to-text format are not suitable for multilingual tasks. |
| Approach: | They propose a unified Transformer framework that converts all language problems into a text-to-text format. |
| Outcome: | The proposed model performs better on all ARGEN tasks than existing models with 49 less data. |
Exploring Label Hierarchy in a Generative Way for Hierarchical Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for hierarchical text classification are lacking in the field of natural language processing. |
| Approach: | They propose a hierarchy-aware T5 model with path-adaptive attention mechanism to exploit hierarchical dependency across different levels. |
| Outcome: | The proposed model outperforms state-of-the-art models especially in Macro-F1 and low Macro. |
Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Popular GEC models use large-scale synthetic corpora or use a large number of human-designed rules. |
| Approach: | They propose a model that incorporates denoised AMR as additional knowledge to get AMRs more reliable. |
| Outcome: | The proposed model reduces training time by 32% while inference time is comparable. |
Multitask Instruction-based Prompting for Fallacy Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Fallacies are used as seemingly valid arguments to support a position and persuade the audience about its validity. |
| Approach: | They propose to use instruction-based prompting to recognize 28 unique fallacies across datasets . they also analyze the effect of model size and prompt choice on model performance . |
| Outcome: | The proposed approach can recognize 28 unique fallacies across domains and genres. |
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2023.findings-acl)
Copied to clipboard
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
| Challenge: | Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model . however, the architecture used for FiD was not designed for retrieval augmented models . |
| Approach: | They propose to make FiD a modified retrieval-augmented language model with a large decoder and memory bandwidth constraints to alleviate memory bandwidth limitations. |
| Outcome: | The proposed architecture outperforms existing models on knowledge-intensive tasks even on large models on many knowledge-based tasks. |
Dual-Stage Multi-Task Syntax-Oriented Pre-Training for Syntactically Controlled Paraphrase Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactically controlled paraphrase generation (SCPG) aims to generate sentences with syntactic structures resembling given exemplars. |
| Approach: | They propose a dual-stage multi-task pre-training scheme that uses a series of structure-oriented and syntax-oriented tasks to generate sentences with syntactic structures resembling given exemplars. |
| Outcome: | The proposed method outperforms existing methods on all possible variants of SCPG tasks and significantly outperformed the popular T5 model. |
Efficient Data Learning for Open Information Extraction with Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the time (3 minutes) to achieve comparable results. |
| Approach: | They propose a framework that transforms OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. |
| Outcome: | The proposed framework transforms OpenIE into the pre-training task form of the T5 model, reducing the need for extensive training data and significantly reducing training time. |